METHOD AND APPARATUS FOR COLLECTING MANAGEMENT 



INFORMATION ON A COMMUNICATION NETWORK 



Background 

1. Field 

[0001] This application relates to communication networks and, more particularly, to a 
method and apparatus for collecting management information on a communication network. 

2. Description of the Related Art 

[0002] Data communication networks may include various computers, servers, nodes, 
routers, switches, hubs, proxies, and other devices coupled to and configured to pass data to one 
another. These devices will be referred to herein as "network elements," and may provide a 
variety of network resources on the communication network. Conventionally, data has been 
communicated through the data conmiunication networks by passing protocol data units (or cells, 
frames, or segments) over links extending between the network devices. A particular protocol 
data unit may be handled by multiple network devices and cross multiple communication links as 
it travels between its source and its destination over the network. 

[0003] Network elements operating on the network may monitor numerous aspects of their 
status, flows passing through the network, and operation of the network itself hideed, a 
conventional network element may be configured to monitor over a thousand different aspects of 
its operation. Management information of this nature is conventionally stored in a Management 
Information Base (MIB) although it may be stored elsewhere on the network element as well. 
Information from the MIB is then conveyed to a management station to enable a network 
operator to monitor the health of the network and network elements operating on the network. 

[0004] Transmission of the thousands or more of aspects of network performance and health 
from the network elements to the management station has several drawbacks. First, as the 
number of monitored attributes on the network elements increases and as the number of network 
elements on the communication network increases, the amount of overhead attributable to 
network management increases, thus limiting scalability of the network. Second, a management 
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Station on the network is typically charged with monitoring the performance and health of a 
number of network elements. As the size of the network elements, and hence the amount of 
management information increases, and as the number of managed network elements increases, 
the volume of information that must be collected and processed increases, which also impacts the 
scalability of the network. Finally, the network elements on a network may be manufactured by 
numerous companies which may be configured to generate different management information. 
Accordingly, in addition to increasing the complexity of the management station software which 
is tasked with interpreting the data received from the managed elements, detailed knowledge of 
the underlying technology is required to extract meaningfiil information from the collected 
management information. 

[0005] Attempts have been made to reduce the overhead on the network and also to reduce 
the complexity of the management information. For example, applications have been created to 
remotely monitor managed entities and cache the results of the queries and make the results 
available to the management application or operator. This method still requires the transmission 
of large amounts of information on the network and has an additional problem that the 
information in the cache may become stale. 

[0006] Another method that has been attempted is to create specialized queries on the 
network elements that monitor specific key measurements for an undesirable behavior. While 
this may alleviate the need to transport all of the management information on the network, it 
requires specialized knowledge of the specific technologies and encounters scalability problems 
with large devices or on large networks. Additionally, attempts have been made to define 
technology neutral health indicators, but these are not universally available. Where the health 
indicators are computed by the network element, the raw data supporting the health indicator is 
not transferred on the network, thus reducing the amount of traffic on the network, but it is not 
always clear how these health indicators relate to the technology specific measurements 
underlying the health indicator. It can therefore be difficult to determine what information to 
retrieve from the network element to ascertain additional information about the state of the 
network element when a health indicator indicates an unhealthy condition. Additionally, where 
the health indicators are computed by the management station, the raw information is required to 
be transmitted on the network which, as indicated above, is undesirable for several reasons. 
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Summary of the Disclosure 

[0007] As described in greater detail herein, a method and apparatus for collecting 
management information on a conmiunication network enables universal health indicators to be 
defined so that management information may be presented to a network operator on a 
management station. The universal health indicators may be defined in a top-down manner such 
that the definitions of the health indicators may be used across multiple network elements or 
network technologies without changing the health indicator definitions. This enables the health 
indicators to be readily interpreted by the network operator. Additionally, the health indicators 
are organized in a tree-like hierarchical structure to enable additional information to be obtained 
about the health of the reporting network element in a predictable and organized fashion. 
Network element raw measurements and computed measurements are used to provide data to the 
health indicators, and are selected on a network element type or MIB type basis to enable the 
same health indicators to be supported by multiple network element types. Additionally, by 
maintaining the raw data on the network element instead of transporting it on the network, the 
amount of management information passed between the network elements and management 
station may be minimized. 

Brief Description of the Drawings 

[0008] Aspects of the present invention are pointed out with particularity in the claims. The 
following drawings disclose one or more embodiments for purposes of illustration only and are 
not intended to limit the scope of the invention. In the following drawings, like references 
indicate similar elements. For purposes of clarity, not every element may be labeled in every 
figure. Ill the figures: 

[0009] Fig. 1 is a functional block diagram of an example of a communication network 
including a management center according to an embodiment of the invention; 

[0010] Fig. 2 is a fimctional block diagram of a hierarchical tree structure used to define a 
health indicator according to an embodiment of the invention; 

[0011] Fig. 3 is a flow diagram illustrating a process of defining health indicators for use in 
the software envirormient of Fig. 2 according to an embodiment of the invention; 
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[0012] Fig. 4 is a functional block diagram of a software environment associated with 
management of network elements on a communication network according to an embodiment of 
the invention 

[0013] Fig. 5 is a functional block diagram of a management station according to an 
embodiment of the invention; and 

[0014] Fig. 6 is a functional block diagram of a network element according to an 
embodiment of the invention. 

Detailed Description 

[0015] The following detailed description sets forth numerous specific details to provide a 
thorough understanding of the invention. However, those skilled in the art will appreciate that 
the invention may be practiced without these specific details. In other instances, well-known 
methods, procedures, components, protocols, algorithms, and circuits have not been described in 
detail so as not to obscure the invention. 

[0016] According to one embodiment of the invention, high-level health indicators may be 
defined based on parameters of interest to a network administrator, and then one or more layers 
of intermediate-level and raw health indicators may be defined to be used to compute the high- 
level health indicators. The health indicators at the various levels may depend on computed 
measurements and raw measurements made by the network elements, which may vary according 
to the type of network technology being monitored. 

[0017] In operation, the health indicators at the various levels may be computed and stored 
by the network element and the high-level health indicators may be communicated to the 
network management station. If one of the high-level health indicators indicates that the network 
element warrants attention by the network operator, additional information about the health 
indicator such as the intermediate-level health indicators or raw health indicators may be 
communicated to the management station to provide details associated with the high-level health 
indicator. Communication of this additional information may be automatic or done on demand. 
In this manner management information may be computed by the network element and 
communicated to the management station in a decipherable fashion, and supporting information 
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may be provided to the management station when required. These and other embodiments will 
be discussed in greater detail below. 

[0018] Fig. 1 illustrates a communication network 10 in which Customer Edge (CE) network 
elements 12 are interconnected over a provider's network containing Provider Edge (PE) 
network elements 14 and Provider (P) network elements 16. Conventionally, PE network 
elements 14 are configured to interface with the CE network elements 12 (which may be owned 
by the provider or owned by the customer) whereas P network elements are configured to 
operate wholly within the provider's network. Optionally, different protocols may be used to 
interface between the interfaces between CE, PE, and P network elements. For example, in an 
optical network, interactions between the CE network elements 12 and the PE network elements 
14 may take place using a suitable optical protocol such as User to Network Interface (UNI), 
although numerous other protocols may be used to control interactions between the CE and PE 
network elements. Exchanges between the PE and P network elements may take place using 
Network to Network Interface (NNI) or another suitable protocol. The invention discussed 
herein is not limited to a particular network configuration or to particular management and 
control protocols in use on the network. Thus, although an embodiment of the invention will be 
described herein in connection with the example network set forth in Fig. 1, the invention is not 
limited to implementation on this type of network or on the particular example network 
illustrated herein. 

[0019] Fig. 1 also includes a management center 18, configured to interface with the P and 
PE network elements, and optionally with the CE network elements. The management center is 
configured to implement a management station on the network to monitor operation of the 
network and take corrective action when warranted on the network. The management center 1 8 
may be located on the provider's network in a convenient location, such as in a control center, or 
may be co-located with one or more of the network elements 14, 16 forming the network 10. 
Optionally, the management center may be instantiated as a distributed process on the network or 
multiple management centers may be included on the network for redundancy purposes, to 
provide a backup facility should something happen to the primary facility or should connectivity 
between one or more of the network elements and the management center be disrupted. The 
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invention is not limited to including the management center at a particular location or plurality of 
locations on the network. 

[0020] The management center may be connected to the network elements using dedicated 
control channels (dashed lines in Fig. 1), or may be connected to the network elements through 
the data network interconnecting the network elements (solid lines in Fig. 1). The invention is 
not limited to a particular manner of interconnecting the management center and the network 
elements. Numerous management protocols have been developed to enable information to be 
obtained from network elements and to enable corrective actions to be taken on network 
elements over the network. For example, several control protocols that may be used include 

• Transaction Language 1 (TLl), a telecommunications management protocol used 
extensively to manage SONET and optical network devices; 

• Simple Network Management Protocol (SNMP), an IETF network monitoring and 
control protocol used extensively to monitor and adjust Management Information Base 
(MIB) values on network devices such as routers and switches; 

• Command Line Interface (CLI), an interface to an application that accepts typed input 
commands; and 

• Common Object Request Broker Architecture (CORBA), a standard from the Object 
Management Group (OMG) for communicating between distributed objects. 

although the invention is not limited to a particular implementation configured to use particular 
protocol(s). 

[0021] The management center monitors the status of the network and collects information 
related to the health of the network and the network elements. According to an embodiment of 
the invention, high level health indicators that are of interest to the management center are 
defined on the network and specified on the network elements, to enable the network elements to 
provide meaningfiil health information to the management center 18. The health indicators may 
be defined in a manner independent of the management center, for example by an applicable 
standard, or may be defined by the management center for the particular network or network 
technologies being managed. 
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[0022] Fig. 2 illustrates a hierarchical tree-like structure of a high-level health indicator 
definition according to an embodiment of the invention. As shown in Fig. 2, the health indicator 
20 includes two parts - a network technology independent portion 22 and a network technology 
specific portion 24. The network technology independent portion 22 is defined by the 
management center or other construct to define the high level health indicator properties and 
supporting intermediate layer health indicators and raw health indicators that may be of interest 
to a network operator when diagnosing a problem on the network. The network technology 
specific portion 24 includes raw measurements and computed measurements, derived fi-om one 
or more raw measurements, that are required to provide information to health indicators. 

[0023] Specifically, as shown in Fig. 2, a health indicator according to an embodiment of the 
invention includes a high level health indicator 26, one or more intermediate computed health 
indicators 28, and raw health indicators 30. The raw health indicators, in this embodiment, are 
derived or computed fi'om raw measurements 32 taken by the network element and/or computed 
measurements 34 that are derived from the raw measurements taken by the network element. In 
the embodiment illustrated in Fig. 2, only one row of intermediate computed health indicators 28 
has been shown. The invention is not limited in this manner, however, as other intermediate 
computed health indicators may be interposed between the high level health indicator and the 
raw health indicators as well. 

[0024] In the embodiment illustrated in Fig. 2, the network element measures values on the 
network, for example the number of packets arriving at an interface in a particular period of time. 
This raw measurement is then passed directly to a raw health indicator or may be passed to a 
computed measurement where it may be manipulated to create another measurement of interest. 
For example, assume that the network element is capable of counting the number of packets 
arriving on an interface. This value may be used by a computed measurement to determine an 
average packet arrival rate on that interface. Multiple raw measurements may be used to create a 
computed measurement. Similarly, multiple raw measiu-ements or computed measurements may 
be passed to a raw health indicator. 

[0025] The raw health indicator receives as inputs the result of one or more raw 
measurements and/or computed measurements, and computes a raw health indicator. One or 
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more raw health indicators may be used as inputs to an intermediate computed heaUh indicator 
and used to compute the value of that health indicator. The intermediate or raw health indicators 
are then used to compute the value of the high level health indicator. 

[0026] Computed health indicators (raw or intermediate) may be combined to form the high 
level health indicator. The high level health indicator may indicate an aspect of interest to the 
network operator, such as the presence of congestion on the network or an alarm state on the 
network element that requires the operator's attention. High level health indicators may be 
computed from raw health indicators, computed health indicators, or a combination of other 
health indicators. 

[0027] When a network operator receives a high level health indicator that indicates a 
problem on the network, the network operator may need additional information about the 
problem. According to an embodiment of the invention, the network operator may obtain this 
information by viewing the computed intermediate and raw health indicators that were used to 
obtain the high level health indicator. For example, assume that a high level health indicator 
indicates congestion on the network. The network operator may obtain the computed health 
indicators that were used to compute the high level health indicator to determine which of the 
several computed health indicators indicated the likely presence of congestion on the network. 
By recursing deeper into the hierarchical tree-like management structure of the health indicator 
the network manager may obtain additional information in an expected manner to ascertain the 
likely cause of the problem to enable corrective action to be taken on the network element. 

[0028] According to an embodiment of the invention, the raw measurements and computed 
measurements are maintained by the network element and only the health indicators are passed 
to the management center. By maintaining the measurement information on the network 
element, the amount of information required to be transmitted on the network and processed by 
the management center may be minimized to allow larger numbers of network elements to be 
managed by a given management center. Optionally, the raw health indicators and intermediate 
computed health indicators may also be maintained on the network element until required by the 
management center to further reduce management traffic on the network. Additionally, by 
organizing the management information in a tree-like structure, the management information 
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may be accessed in a logical manner thus simplifying the diagnostic process associated with 
managing network elements on a network. Particularly, since the same hierarchical structure 
may be used to manage many different types of network technologies, advanced knowledge of 
the particular network element generating the adverse health indicator is not required of the 
network operator so that the network operator may diagnose problems on the network more 
easily. 

10029] Fig. 3 illustrates a process of defining high level health indicators to enable network 
elements to communicate management information on the network. As shown in Fig. 3, instead 
of starting with the raw measurements that are available on the network element and deriving 
from the available measurements various information, the process begins by defining a short 
series of high-level computed health indicators that will be of interest to the network manager 
while managing the network (ICQ). Examples of several high-level computed health indicators 
may be network element failure, link failure, congestion, contention, and various other indicators 
that may signal the presence of a problem on the network, for example the inability of the 
network to provide sufficient resources to meet a subscriber's service level agreement. The 
invention is not limited to these particular examples, however, as the health indicators may be 
used to indicate many conditions on the network. 

[0030] Once the high-level computed health indicators are defined, the process will 
determine, for each high level health indicator, what intermediate computed health indicators 
will be required to compute that high level health indicator (102), and the computations that will 
be required to derive the associated high level health indicator. Then, for each high level or 
intermediate health indicator, the process will determine what raw health indicators will be 
required to compute the high or intermediate level health indicator (104) as well as the 
computations that will be required. As discussed above, the intermediate and raw health 
indicators may be selected to provide a hierarchical tree structure that may be of use to a network 
manager attempting to trace down the source of the problem associated with a high level health 
indicator. 

[0031] Once the health indicator(s) are defined for use on the management center, the health 
indicators need to be tailored to accommodate the measurement capabilities of the network 
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technologies. Accordingly, for each network technology type that will be monitored, the health 
indicators are parsed to determine what computed measurements (106) and raw measurements 
(108) are required to support the defined health indicator. In making this determination, the 
process assesses what raw measurements are available, and what information is needed to be 
provided to the raw health indicators. This process enables the high level and other intermediate- 
level health indicators to be tied to available measurements, which may vary depending on the 
type of network technology, manufacturer, management information base, and other aspects of 
the network element's design. 

[0032] By initially defining the high level health indicator and tree structure supporting the 
high level health indicator without reference to the available raw measurements, it is possible to 
use the same high level health indicator generically across network elements regardless of the 
type of network technology and other contributing factors that may affect network management 
appearance. By breaking the high level health indicator into a plurality of supporting raw health 
indicators, it is possible to more easily mate the health indicator with the raw measurements and 
computed measurements available on the specific network elements. 

[0033] Fig. 4 illustrates a software control package that may be implemented on the 
management center 18 and on one or more of the network elements to implement an embodiment 
of the invention. As shown in Fig. 4, the software control package includes a management 
center software environment 40 including a user interface 42 configured to enable the 
management center to interface a network manager or other client interface associated with a 
network management program. For example, the user interface may include a graphical user 
interface, command line interface, web browser interface, or other interface to enable a network 
manager to view information about the state or status (e.g. health indicators) of the network and 
network elements on the network. 

[0034] The management center software environment also includes a management protocol 
interface 44 to enable it to communicate with network elements on the network and a control 
software module 46 to enable the network operator to take corrective action on the network. The 
management center software environment may also include additional software modules such as 
a network topology 48 module configured to enable the network manager to have a view of the 
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network and a health monitor software module 50 to enable the network manager to monitor the 
health indicators on the network. Optionally, the management center software environment may 
include a health definition creation software module 52 configured to create health indicator 
definitions and a store of available high level health indicator definitions 54. The invention is 
not limited to this embodiment, however, as the health definitions (including high-level health 
indicator definitions) may be created external to the management center software environment. 
Optionally, a secvirity module may be provided to enable network elements and the network 
management center to operate in a relatively secure environment, although the invention is not 
limited in this manner. 

[0035] The software control package fiirther includes a network element control software 
environment 60 including a management protocol interface 62 to enable the network element to 
engage in transactions with the management center, and network element component interfaces 
64 to enable the network element to interface internally with the components of the network 
element to obtain the required raw measurement information. 

[0036] A health definition software module 66 is provided to enable the network element to 
calculate computed measurements and health indicators according to health indicator definitions 
being used by the management station to monitor the network element. Optionally, a 
management information base 68 may be provided to store the raw measurements, and optionally 
computed measurements, associated with the network element. 

[0037] When a raw measurement changes, the new value may be stored in the MIB. This 
change may affect one or more computed measurements, which in turn may affect the various 
computed health indicators. The timing and fi^equency at which the changes to raw 
measurements are used to update computed measurements and health indicators is a matter of 
policy, and hence may be implemented in a number of different ways. The invention is not 
limited to any particular manner of updating the health indicators based on changes to the raw 
measurements. 

[0038] Fig. 5 illustrates an embodiment of a management center 18 according to an 
embodiment of the invenfion. As shown in Fig. 5, the management station may be configured to 
be implemented on a network element including a processor 70 having control logic 72 
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configured to implement the functions ascribed to the management station described herein. 
Alternatively, the management station may be instantiated on a network element providing other 
services on the network and may be configured to run on the processor and control logic of the 
host network element. One or more input/output ports 74 may be provided to interface the 
management station to the network to enable it to communicate with network elements on the 
network. 

[0039] The management center may also include management control software environment 
40 discussed above in connection with Fig. 4, and configured to enable management of the 
conmiunication network via monitoring of high level health indicators and, when necessary, 
obtaining additional details about particular aspects of the network element via the intermediate 
and raw computed health indicators. Optionally, a database of network element health 
information 76 may be maintained as well, although the invention is not limited to an 
embodiment that includes this additional store of information.. 

[0040] The management center may also include a network policy module 78 responsible for 
obtaining and setting policy information on the network. The policy information may affect 
when a particular high-level health indicator will be presented to a network operator, values used 
in the calculation to obtain particular high-level or intermediate health indicators, and otherwise 
to affect the presentation of management information to the network operator. For example, a 
particular network operator may wish to receive a health indication when a network element's 
traffic level exceeds 75% of its capacity while another network operator may not wish to receive 
that health indicator until the traffic level exceeds 90%. This preference may be implemented 
via the policy module 78 on the management center. 

[0041] Although several specific modules have been described herein in connection with the 
management center, the invention is not limited to an embodiment that implements all of these 
modules or only these modules, as the management center may be implemented in myriad other 
ways without departing from the scope of the invention. 

[0042] Fig. 6 illustrates a network element configured to implement an embodiment of the 
invention. As shown in Fig. 6, the network element includes a plurality of input/output (I/O) 
cards 80 configured to be connected to one or more types of physical media. Optionally, the I/O 
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cards may also include processing capabilities to enable them to perform preliminary processing 
on packets received over the physical media. Packets received are passed to a network processor 
82 having control logic 84 which performs necessary processing operations on the packets and 
passes them to a switch fabric 86. The switch fabric enables the packets to be passed to another 
network processor which, upon completion of its processing tasks, passes the packets to one of 
the I/O cards to be output on a physical media. The particular intemal structure of the network 
element is not intended to limit the scope of the invention as numerous network element 
architectures may be used. 

[0043] The network processors, I/O cards, switch fabric, and numerous other modules on the 
network element may be configured to measure values and pass the measured values to a control 
interface 86 on the network element. The control interface 86 may be implemented on one or 
more of the network processors 82 or may be instantiated in a separate processing environment, 
such as in a dedicated processor on the network element. The control interface 86 includes the 
network element control software environment 60 described above in connection with Fig. 4, 
which includes, for example, the health definition software 66 configured to enable the network 
element to calculate computed measurements and health indicators according to health indicator 
definitions being used by the management station to monitor the network element. The control 
interface 86 may also include other software modules, such as an interface manager 90, protocol 
stack 92, and statistics collection module 94, configured to support the modules in the control 
software environment. The invention is not limited in this manner, however, as other groups of 
modules may be provided to compliment the fiinctionality described above with respect to the 
network element control software environment. 

[0044] The control logic 72, 84 may be implemented as a set of program instructions that 
are stored in a computer readable memory within or interfaced to the network element and 
executed on a microprocessor, such as processor 70,82. However, in this embodiment as with 
the previous embodiments, it will be apparent to a skilled artisan that all logic described herein 
can be embodied using discrete components, integrated circuitry such as an Application Specific 
Integrated Circuit (ASIC), programmable logic used in conjunction with a programmable logic 
device such as a Field Programmable Gate Array (FPGA) or microprocessor, or any other device 
including any combination thereof Programmable logic can be fixed temporarily or 
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permanently in a tangible medium such as a read-only memory chip, a computer memory, a disk, 
or other storage medium. Programmable logic can also be fixed in a computer data signal 
embodied in a carrier wave, allowing the programmable logic to be transmitted over an interface 
such as a computer bus or communication network. All such embodiments are intended to fall 
within the scope of the present invention. 

[0045] It should be understood that various changes and modifications of the embodiments 
shown in the drawings and described herein may be made within the spirit and scope of the 
present invention. Accordingly, it is intended that all matter contained in the above description 
and shown in the accompanying drawings be interpreted in an illustrative and not in a limiting 
sense. The invention is limited only as defined in the following claims and the equivalents 
thereto. 

[0046] What is claimed is: 
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